# Multimodal Model

Awesome GPT-4o Images is a collection showcasing images and prompts generated by OpenAI's latest multimodal model, GPT-4o. This product fully demonstrates GPT-4o's capabilities in text and image understanding, supporting the generation of various art styles. It is suitable for designers, artists, and anyone interested in AI art. This project is free and open-source, aiming to inspire creativity and promote the development of AI art.

TableGPT-agent is a pre-built agent model based on TableGPT2, designed for question-answering tasks involving tabular data. Developed using the Langgraph library, it offers a user-friendly interface and efficiently handles complex table-related questions. TableGPT2 is a large multimodal model that combines tabular data with natural language processing, providing powerful support for data analysis and knowledge extraction. This model is suitable for scenarios requiring fast and accurate processing of tabular data, such as data analysis, business intelligence, and academic research.

Aria-UI is a large-scale multimodal model specifically designed for visual localization of GUI commands. It employs a purely visual approach without relying on auxiliary inputs, accommodating a variety of planning commands and generating diverse, high-quality command samples to adapt to different tasks. Aria-UI has set new records in both offline and online agent benchmarks, surpassing baselines that rely solely on visual inputs or AXTree.

OmniAudio-2.6B is a multimodal model with 2.6 billion parameters that seamlessly processes both text and audio inputs. This model combines Gemma-2B, Whisper Turbo, and a custom projection module. Unlike the traditional method of chaining ASR and LLM models, it unifies both capabilities in an efficient architecture, achieving minimal latency and resource overhead. This enables it to securely and rapidly process audio-text directly on edge devices such as smartphones, laptops, and robots.

ultravox-v0_4_1-mistral-nemo is a multimodal speech large language model (LLM) based on pre-trained Mistral-Nemo-Instruct-2407 and whisper-large-v3-turbo. The model can handle both speech and text input simultaneously, such as a text system prompt and a speech user message. Ultravox converts input audio into embeddings using a special <|audio|> pseudo-token and generates output text. Future versions plan to expand the token vocabulary to support generating semantic and acoustic audio tokens, which can then be input into a vocoder to produce speech output. The model is developed by Fixie.ai and is licensed under MIT.

fixie-ai/ultravox-v0_4_1-llama-3_1-8b is a large language model based on pre-trained Llama3.1-8B-Instruct and whisper-large-v3-turbo, capable of processing speech and text input to generate text output. The model converts input audio to embeddings using a special <|audio|> pseudo-token and generates output text. Future versions plan to expand the token vocabulary to support semantic and acoustic audio token generation, which can then be used by a vocoder to produce speech output. The model performs excellently in translation evaluation and has no preference adjustment, making it suitable for scenarios such as voice agents, speech-to-speech translation, and speech analysis.

TableGPT2 is a large multimodal model specifically pre-trained and fine-tuned for tabular data to address the challenges of inadequate integration in practical applications. It was pre-trained and fine-tuned on over 593.8K tables and 2.36M high-quality query-table-output tuples, achieving unprecedented scale. A key innovation of TableGPT2 is its novel table encoder, designed to capture information at both pattern and cell levels, enhancing the model's ability to handle ambiguous queries, missing column names, and irregular tables. On 23 benchmark metrics, the average performance of TableGPT2 improved by 35.20% for the 7B model and by 49.32% for the 72B model, while maintaining robust general language and coding capabilities.

MiniGPT4-Video is a multimodal large model designed for video understanding. It can process temporal visual data and text data, generate captions and slogans, and is suitable for video question answering. Based on MiniGPT-v2, it incorporates the visual backbone EVA-CLIP and undergoes multi-stage training, including large-scale video-text pre-training and video question-answering fine-tuning. It achieves significant improvements on benchmarks such as MSVD, MSRVTT, TGIF, and TVQA. The pricing is currently unknown.

Bubble Duck AI is an AI interaction platform developed based on self-developed multimodal large models, providing a massive number of intelligent agents in various domains such as anthropomorphism, tools, content, games, and entertainment. The platform has a super-long context memory capability and real-time network search capability, enabling it to deeply understand user intent and provide instant, accurate, and personalized responses and choices. Users can also customize personalized AI intelligent agents to meet individual needs and preferences.

WebVoyager is an innovative large multimodal model (LMM)-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. We propose a novel web agent evaluation protocol to address the challenge of automatic evaluation for open-world agent tasks, leveraging the powerful multimodal understanding capabilities of GPT-4V. We collected real-world tasks from 15 widely used websites to evaluate our agent. We demonstrate that WebVoyager achieves a 55.7% task success rate, significantly outperforming the performance of GPT-4 (with all tools) and WebVoyager (text only) settings, highlighting WebVoyager's superior capabilities in practical applications. We find that our proposed automatic evaluation achieves 85.3% consistency with human judgment, paving the way for further development of web agents in real-world environments.